Synthesizing larger texture images from a smaller exemplar is an important task in graphics and vision. The conventional CNNs, recently adopted for synthesis, require to train and test on the same set of images and fail to generalize to unseen images. This is mainly because those CNNs fully rely on convolutional and upsampling layers that operate locally and not suitable for a task as global as texture synthesis. In this work, inspired by the repetitive nature of texture patterns, we find that texture synthesis can be viewed as (local) \textit{upsampling} in the Fast Fourier Transform (FFT) domain. However, FFT of natural images exhibits high dynamic range and lacks local correlations. Therefore, to train CNNs we design a framework to perform FFT upsampling in feature space using deformable convolutions. Such design allows our framework to generalize to unseen images, and synthesize textures in a single pass. Extensive evaluations confirm that our method achieves state-of-the-art performance both quantitatively and qualitatively.

Unless otherwise stated, the evaluation setup mimics the setup adopted in the main paper. Note, fast Fourier transform (FFT) is an efficient implementation of DFT. The first two authors equally contributed. There is no gold standard image quality metric, and most previous work on texture synthesis only provide visual comparison. Table I below lists the average quality metrics for 5,000 test examples. It cannot directly take the small 128x128 texture patch as the input.

The rebuttal is helpful in clarifying concerns and making the final decision. Here are my final suggestions to improve the draft: 1. The description in Line 4-10 of rebuttal should go somewhere in Sec-1. It is more helpful than Line 25-36 in current submission draft. It would be highly informative to the reader if the authors could include the discussion in main paper or appendix.

All four expert reviewers agree on the merit of the submission. The authors are encouraged to incorporate the suggestions of the reviewers into the final version.

Synthesizing larger texture images from a smaller exemplar is an important task in graphics and vision. The conventional CNNs, recently adopted for synthesis, require to train and test on the same set of images and fail to generalize to unseen images. This is mainly because those CNNs fully rely on convolutional and upsampling layers that operate locally and not suitable for a task as global as texture synthesis. In this work, inspired by the repetitive nature of texture patterns, we find that texture synthesis can be viewed as (local) \textit{upsampling} in the Fast Fourier Transform (FFT) domain. However, FFT of natural images exhibits high dynamic range and lacks local correlations.